Papers with reinforcement learning (
Inverse Reinforcement Learning Meets Large Language Model Alignment (2025.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide a comprehensive review of recent advances in LLM alignment . it will highlight the necessity of constructing neural reward models from human data . |
| Approach: | This tutorial will provide a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning. |
| Outcome: | This tutorial will provide a comprehensive review of recent advances in LLM alignment through the lens of inverse reinforcement learning (IRL). |
DeepTrans: Deep Reasoning Translation via Reinforcement Learning (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown promising performance in various downstream tasks. |
| Approach: | They propose a deep reasoning translation model that learns free translation via reinforcement learning (RL) they build a reward model with pre-defined scoring criteria on the translation results and thought processes . |
| Outcome: | The proposed model outperforms strong deep reasoning LLMs in literature translation and outperformed other models. |
[CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in off-policy reinforcement learning methods that use offline data as against a simulator have proven to be sample efficient. |
| Approach: | They propose a batch-RL framework for ToD policy learning: Causal-aware Safe Policy Improvement (CASPI) that uses a mechanism to learn fine-grained reward that captures intention behind human response and offers guarantee on dialogue policy’s performance against a baseline. |
| Outcome: | The proposed framework outperforms the current state of the art on an end-to-end dialogue task using a multiwoz2.0 dataset. |
Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning (2022.coling-1)
Copied to clipboard
| Challenge: | In task-oriented dialogue systems, the role of the natural language generation component is to convert a system's intentions, called dialogue acts (DAs), into natural language utterances and to convey DAs accurately to users. |
| Approach: | They propose a method for Adaptive Natural language generation for Task-Oriented dialogue via Reinforcement learning that incorporates a natural language understanding module into the objective function of RL. |
| Outcome: | The proposed method generates adaptive utterances against speech recognition errors and the different vocabulary levels of users in a multi-world task-oriented dialogue system. |
Reward Engineering for Generating Semi-structured Explanation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Unstructured natural language explanations lack a comprehensive explanation mechanism to verify a model's true reasoning capabilities. |
| Approach: | They propose a reward engineering method which uses semi-structured explanations to verify a model's true reasoning capabilities. |
| Outcome: | The proposed method achieves new state-of-the-art on two semi-structured explanation generation benchmarks (ExplaGraph and COPA-SSE) . |
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. |
| Approach: | They propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. |
| Outcome: | The proposed model improves response precision while preserving exploratory ability to uncover potential correct pathways. |
Rewarding Semantic Similarity under Optimized Alignments for AMR-to-Text Generation (2022.acl-short)
Copied to clipboard
| Challenge: | Automatic evaluation metrics score natural language generation systems based on how well they lexically align to humanannotated references. |
| Approach: | They propose to replace greedy alignments in BERTScore with optimized ones that replace the n-gram matching BERTAcore metrics with a token embedding to prevent domain mismatch. |
| Outcome: | The proposed metrics outperform cross-entropy and BLEU reward baselines on AMR-to-text generation. |
Efficient Text-based Reinforcement Learning by Jointly Leveraging State and Commonsense Graph Representations (2021.acl-short)
Copied to clipboard
Keerthiram Murugesan, Mattia Atzeni, Pavan Kapanipathi, Kartik Talamadupula, Mrinmaya Sachan, Murray Campbell
| Challenge: | Text-based games (TBGs) are useful benchmarks for evaluating progress in grounded language understanding and reinforcement learning (RL). |
| Approach: | They propose an agent that induces a graph representation of the game state and jointly grounds it with a commonsense knowledge from ConceptNet. |
| Outcome: | The proposed agent outperforms baseline agents in the proposed game . |
Scheduled Dialog Policy Learning: An Automatic Curriculum Learning Framework for Task-oriented Dialog System (2021.findings-acl)
Copied to clipboard
| Challenge: | et al., 2013) show that dialog policy learning is an important component of the task-oriented dialogue system. |
| Approach: | They propose a framework that integrates curriculum learning and policy optimization . they propose to train dialog agents from easy dialogues to complex ones . |
| Outcome: | The proposed framework outperforms the state-of-the-art model on multi-task dialogues. |
Reinforcement Learning with Token-level Feedback for Controllable Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for controllable text generation are guided by coarse-grained feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. |
| Approach: | They propose a reinforcement learning algorithm which formulates TOken-LEvel rewards for controllable text generation and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm. |
| Outcome: | The proposed algorithm can achieve superior performance on single-attribute and multi-attract control tasks. |
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (P18-1)
Copied to clipboard
| Challenge: | Recent work has shown that reinforcement learning (RL) can be scaled to games with large state-action spaces, achieving human-level performance or even superhuman performance. |
| Approach: | They propose to use bandit feedback to improve sequence-to-sequence learning by simulating reward signals by evaluation metrics such as BLEU, F1-score, or ROUGE. |
| Outcome: | The proposed methods improve performance even from small amounts of human feedback, pointing to a great potential for applications at larger scale. |
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for embodied agents to learn and perform tasks use low-level instructions, which may not reflect natural human communication. |
| Approach: | They propose to use different types of language inputs to facilitate reinforcement learning (RL) embodied agents. |
| Outcome: | The proposed methods show that agents trained with diverse and informative language can achieve enhanced generalization and fast adaptation to new tasks in an open world. |
StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback (2024.acl-long)
Copied to clipboard
Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang
| Challenge: | Existing work integrates reinforcement learning with compiler feedback to enhance code generation quality but the long code generated by LLMs makes RL exploration ineffective. |
| Approach: | They propose a framework that integrates reinforcement learning and compiler feedback to enhance code generation quality. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches in corresponding benchmarks and integrates reinforcement learning with compiler feedback to improve code generation quality. |
Learning Autonomous Driving Tasks via Human Feedbacks with Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems focus on making autonomous driving decisions without human interaction, but human-like decision-making is still an important factor in designing autonomous driving systems. |
| Approach: | They propose a framework leveraging Large Language Models for learning human-centered driving decisions from diverse simulation scenarios and environments that incorporate human feedback. |
| Outcome: | The proposed framework can match baseline extensively trained reinforcement learning models in driving scenarios and store optimal driving programming policy using Retrieval Augmented Generation (RAG). |
A Multi-Document Coverage Reward for RELAXed Multi-Document Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Multi-document summarization models are limited by limited references and with maximum-likelihood objectives. |
| Approach: | They propose to fine-tune an MDS baseline with a reward that balances a reference-based metric such as ROUGE with coverage of the input documents. |
| Outcome: | The proposed model improves on the Multi-News and WCEP datasets with a low-variance estimator . the proposed model also improves the coverage of the input documents . |
Revisiting the Roles of “Text” in Text Games (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that random text hashes could be complementary rather than contrasting in text games. |
| Approach: | They propose a scheme to extract contextual information into an approximate state hash as extra input for an RNN-based text agent. |
| Outcome: | The proposed scheme achieves competitive performance with state-of-the-art text agents using advanced NLU techniques such as knowledge graph and passage retrieval. |
Cache-Efficient Posterior Sampling for Reinforcement Learning with LLM-Derived Priors Across Discrete and Continuous Domains (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used as action proposers in reinforcement learning (RL) but they are expensive and require prohibitive computational costs. |
| Approach: | They propose a cache-efficient framework for Bayesian RL that leverages large language models as action proposers and optimizes meta-learning based on policy performance to enable efficient inference across text-based games and robotic control tasks. |
| Outcome: | The proposed framework achieves 3.8–4.7 reduction in LLM queries and 4.0–12.0 lower median latencies (85–93ms on consumer hardware) while retaining 96–98% of the uncached policy’s performance. |
ACECODER: Acing Coder RL via Automated Test-Case Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Recent coder models have been driven by supervised fine-tuning, but the potential of reinforcement learning remains unexplored due to the lack of reliable reward data/model in the code domain. |
| Approach: | They propose a pipeline that generates extensive test-case pairs from existing code data and constructs preference pairs based on pass rates over sampled programs. |
| Outcome: | The proposed pipeline generates extensive (question, test-cases) pairs from existing code data and trains them with Bradley-Terry loss. |
Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for reinforcement learning for large language models do not accurately assess generalization. |
| Approach: | They propose three core principles for designing more faithful benchmarks: sufficient difficulty, balanced evaluation, and distributional robustness. |
| Outcome: | The proposed benchmarks do not accurately assess generalization across distribution shifts, difficulty levels, and counterfactual scenarios. |
Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem Proving (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent methods for AI reasoning require applying variants of reinforcement learning (RL) on rolled out trajectories, even for step-wise rewards, or large quantities of human-annotated trajectory data. |
| Approach: | They propose a verifier-in-the-loop design that uses an automated verifier to give intermediate feedback at each step of the reasoning process. |
| Outcome: | The proposed model improves on the Automatic Theorem Proving task using Lean as the verifier. |
Targeted Exploration via Unified Entropy Control for Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for group relative policy optimization suffer from entropy collapse . Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain stability. |
| Approach: | They propose a framework that provides targeted mechanisms for exploration and stabilization. |
| Outcome: | The proposed framework expands search space on difficult prompts while preventing entropy growth uncontrollably. |
A Comprehensive Survey on Learning from Rewards for Large Language Models: Reward Models and Learning Strategies (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent developments in Large Language Models have shifted from pre-training to post-training and test-time scaling. |
| Approach: | They present a comprehensive overview of learning from rewards from the perspective of reward models and learning strategies across training, inference, and post-inference stages. |
| Outcome: | The proposed paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. |
EMO-RL: Emotion-Rule-Based Reinforcement Learning Enhanced Audio-Language Model for Generalized Speech Emotion Recognition (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in reinforcement learning (RL) have shown promise in improving LALMs’ reasoning abilities, but their performance in affective computing tasks remains suboptimal. |
| Approach: | They propose a framework incorporating reinforcement learning with two key innovations: Emotion Similarity-Weighted Reward (ESWR) and Explicit Structured Reasoning (ESR). |
| Outcome: | The proposed framework improves LALMs' reasoning abilities on MELD and IEMOCAP datasets and shows strong generalization. |
Mixing Inference-time Experts for Enhancing LLM Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving reasoning quality in large language models are limited to using a single expert. |
| Approach: | They propose a framework that finetunes and merges expert logits from one LLM . they use commonsense and entailment reasoning experts to improve chain-of-thought reasoning . |
| Outcome: | The proposed framework outperforms baselines on three question-answering datasets. |
Reinforcement Learning for Large Language Models via Group Preference Reward Shaping (2025.emnlp-main)
Copied to clipboard
Huaisheng Zhu, Siyuan Xu, Hangfan Zhang, Teng Xiao, Zhimeng Guo, Shijie Zhou, Shuyue Hu, Vasant G. Honavar
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) are expensive and sensitive to reward model quality. |
| Approach: | They propose a method that leverages preference-based comparisons rather than precise numerical rewards. |
| Outcome: | Experiments show that GPRS outperforms critic-model-free RL algorithms on RLHF and reasoning tasks. |
StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has demonstrated unprecedented capabilities in sophisticated linguistic comprehension and generative tasks. |
| Approach: | They propose a framework for search LLMs that trains with step-wise proximal policy optimization method to improve QA performance. |
| Outcome: | The proposed framework outperforms global-reward benchmarks on multi-hop QA with a stepwise proximal policy optimization method and richer and more detailed intermediate search rewards and token-level process supervision. |
Rethinking the Role of Proxy Rewards in Language Model Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Typically, the human feedback is used to train a proxy reward model (RM), and a policy model is optimized over the reward signal from the RM using RL. |
| Approach: | They aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals after training the model using the proxy reward in reinforcement learning (RL). |
| Outcome: | The proposed model shows competitive performances with strong open-source RMs in alignment benchmarks. |
GeoPQA: Bridging the Visual Perception Gap in MLLMs for Geometric Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in reinforcement learning (RL) have enhanced the reasoning abilities of large language models, but the impact on multimodal LLMs is limited. |
| Approach: | They propose a two-stage RL framework that enhances visual perception and fosters reasoning capabilities. |
| Outcome: | The proposed framework improves geometric reasoning by 9.7% and problem-solving by 9.1% compared to direct reasoning training approach. |
Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning (2026.acl-long)
Copied to clipboard
Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhong-Zhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang, Zhenfei Yin, Philip Torr, Lei Bai
| Challenge: | elucidating scaling laws for large language models (LLMs) during pre-training remains unexplored. |
| Approach: | They characterize how model scale, data, and compute interact during pre-training . they find that large models consistently demonstrate superior compute and data efficiency . |
| Outcome: | The proposed scaling laws offer practical guidance for scaling reasoning capabilities through reinforcement learning post-training. |
Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) and AI systems have led to a paradigm shift in the design and optimization of complex workflows. |
| Approach: | They propose a systematic review of recent progress in optimizing compound AI systems . they formalize the notion of compound AI system optimization and classify existing methods along several key dimensions . |
| Outcome: | The proposed methods outperform existing methods in the field of compound AI and highlight open research challenges and future directions. |